The Impact of Overfitting and Overgeneralization on the Classification Accuracy in Data Mining

he Impact of Overfitting and Overgeneralization on the Classification Accuracy in Data Mining Huy Nguyen Anh Pham and Evangelos Triantaphyllou Department of Computer Science 298 Coates Hall, Louisiana State University Baton Rouge, LA 70803 Email: [email protected] and [email protected] Webpage: http://www.csc.lsu.edu/trianta Many classification studies often times conclude with a summary table which presents erformance results of applying various data mining approaches on different datasets. No single ethod outperforms all methods all the time. Furthermore, the performance of a classification ethod in terms of its false-positive and false-negative rates may be totally unpredictable. ttempts to minimize any of the previous two rates, may lead to an increase on the other rate. If he model allows for new data to be deemed as unclassifiable when there is not adequate nformation to classify them, then it is possible for the previous two error rates to be very low but, t the same time, the rate of having unclassifiable new examples to be very high. The root to the bove critical problem is the overfitting and overgeneralization behaviors of a given classification pproach when it is processing a particular dataset. Although the above situation is of undamental importance to data mining, it has not been studied from a comprehensive point of iew. Thus, this chapter analyzes the above issues in depth. It also proposes a new approach alled the Homogeneity-Based Algorithm (or HBA) for optimally controlling the previous three rror rates. This is done by first formulating an optimization problem. The key development in his chapter is based on a special way for analyzing the space of the training data and then artitioning it according to the data density of different regions of this space. Next, the lassification task is pursued based on the previous partitioning of the training space. In this way, he previous three error rates can be controlled in a comprehensive manner. Some preliminary omputational results seem to indicate that the proposed approach has a significant potential to ill in a critical gap in current data mining methodologies. eywords: Data mining, classification, prediction, overfitting, overgeneralization, false-positive, alse-negative, homogenous set, homogeneity degree, optimization.